This invention relates to optical correlators and more particularly to a method and apparatus for improving the frame rate to permit real time processing in which a sample image is compared to a number of reference images.
In general, optical correlations have been performed using van der Lugt optical correlators. These correlators utilize spatial light modulators to be able to present for correlation a large number of reference images to which a sampled image is to be compared.
The problem with the prior art optical correlators is the speed with which the reference images can be presented and compared with the sample image. For liquid crystal spatial modulators the frame rate is slow at 10 kHz. Unfortunately, a 10 kHz spatial light modulator is much too slow for real time optical correlator applications. Note that such optical correlators have been devised by Dr. Christina Johnson and Dr. Jim Kirsch.
Even slower are magneto-optical devices which operate at 500 Hz and are intolerably slow. The reason for the slowness of a magneto-optical device is the high voltage swings, in terms of kilovolts, in order for the modulator to switch from one state to another.
Were fast optical comparators realizable, then spatial pattern matching of a fine grain nature could be accomplished in real time. Such optical comparators would have immediate application in detecting and locating defects in semiconductor devices in which it is useful to inspect many areas on a semiconductor chip or substrate to compare them against a xe2x80x9cknown goodxe2x80x9d. This is particularly useful for inspection of field programmable gate arrays, FPGAs, and digital signal processors or DSPs which are embedded processors. In general, spatial pattern matching is exceptionally useful for any electronic chips including DSPs, ASICS, wafer level devices, die level devices and memory. The inspection done optically is used to detect where there is a deviation from a reference and where it is located. A catalog is then generated of expected defects. If one can perform 300,000 correlations per second, one can identify many potential defects in a large expanse of semiconductor substrate.
Optical pattern recognition is also extremely important in the medical field. Most importantly real time optical pattern recognition is a requirement for mammography, pap smear analysis, nuclear magnetic residence or NMR images and DNA mapping involving recognizing the pattern of genes. It is important, for instance, to be able to recognize the number of cancer cells in a pap smear or artifacts in mammography with a high degree of reliability. Spatial pattern matching is an exceedingly accurate way of determining the results of a mammography or pap smear.
However, for all of the above applications the number of reference images to be presented for comparison is quite large. Were optical pattern recognition to be performed through the utilization of the prior liquid crystal techniques, days rather than minutes would be required due to the inability to change reference images quickly.
Another application for real time spatial pattern matching is in the security area in which both face and finger print recognition is required. Here the number of pixels analyzed for a match is so large that any system which is not real time would not be practical. Certainly the slow optical recognition now available would preclude its use in ATM machines or at any place where the public is to be granted access such as to a building, room or other secure area.
Additionally, there is the field of signal identification. It is indeed possible to identify radar, sonar, voice, RF signatures and other types of electromagnetic signals through the utilization of optical correlation techniques in which signals are compared to known reference signals that are encoded as optical images. Identification of friend or foe (IFF) and sonar trace matching are some of the military applications for which optical comparators have major application, assuming real time capability.
Another application for real time optical processing is in hyper spectral imaging. Spectral information for each pixel is sometimes available for data pixels in which one of the variables is wavelength. Thus the frequency or wavelength of the particular data pixel can be ascertained in one embodiment through optical correlation techniques, assuming such correlations could be done in a very rapid fashion. Moreover, spatial frequency detection in images, such as for instance the detection of edges in a photograph, or noise can be an application for optical correlators. In both of the above cases identifying the temporal frequency and/or spatial frequency of a data point in the form of a pixel through the utilization of matching with reference images is an exceedingly accurate way of defining matches.
Thus, whether optical processing is utilized in the checking of semiconductor arrays, for medical uses, for security, or for signal data stream or image analysis, there is a requirement for the ability to present large numbers of reference images to which a sample image is to be correlated.
By way of further background, in spite of remarkable advances in integrated circuit technology over the last three decades, for the most part, calculations are still done serially. Consequently, the capability of implanting significant signal processing capabilities that are intrinsic to many optical systems has remained a major challenge. In some cases, for example with the implementation of Fast Fourier Transform (FFT) methods, specialized chips have been developed that significantly speed up the calculations. However, nothing compares with the massive parallel processing capability associated with the speed with which a Fourier transform can be obtained optically.
One particular application of optical signal processing, and one in which the current technology has proven only modestly successful is that of optical correlation. Optical correlation is a process that uses optical methods to determine whether two images are similar. There is a very large market to be tapped into if a versatile optical correlator can be developed.
One problem which needs to be solved is how to enable real time correlation of objects in optical image correlators.
Unless the orientation of objects in the images being subjected to an optical correlation measurement is almost exactly the same, two images of the exact same object will not appear to be of the same object. In other words, the orientation can only differ by a few degrees. This restriction means that an image of a given object must be compared to a great many images of the reference object to which it is being compared before a good correlation can be found. A large number of images is required because of the need to have views of the object from as many orientations as possible. These comparisons require a large number of calculations, so significant computational speed is needed to complete the correlation calculation for each orientation in a reasonable time.
Existing optical image correlators are so slow that only a few orientations of an image can be correctly correlated per second for complex images. This slow speed dramatically limits the potential of optical image correlators to meet the demands of systems that need optical correlation capability applied to objects that are oriented in arbitrary directions. What is needed is a way of enabling real time correlation of objects in an optical image correlator.
The question then becomes how to implement an optical image correlator that is significantly faster than correlators made with liquid crystals.
A key component of an optical image correlator is a spatial light modulator (SLM). A spatial light modulator is a device that has a two-dimensional array made up of pixels. Each pixel can be controlled to allow one to modulate the reflectivity or phase, or both of an incident beam. By varying the driving voltage and making a phase grating, the phase can be modulated. Thus, by applying a voltage gradient across a spatial light modulator, the beam can be steered. By modulating the reflectivity on a pixel-by-pixel basis, a diffraction grating can be created. By combining the two, a diffraction pattern can be steered. The speed at which this capability can be implemented is very important.
Current spatial light modulators are made predominately of liquid crystals, which are long-chain organic molecules. Their response is slow, but satisfactory for simple display purposes. Their bandwidth is typically a couple of hundred hertz, and at most approximately 1 kHz. However, liquid crystal arrays with bandwidths as high as 10 kHz may be possible. Unfortunately, even a 10 kHz spatial light modulator is much too slow for real time applications of optical image correlators. What is needed is a way of implementing a correlator that is significantly faster than correlators made using liquid crystals or CCD arrays.
Note, U.S. Patents relating to optical correlators include U.S. Pat. Nos. 5,488,504; 5,883,743; 5,659,637; 5,276,636; and 5,900,624.
In the subject invention, the way in which real time capability is achieved in optical correlation is to provide that the spatial light modulator be in the form of a multiple quantum well device. Such a multiple quantum well device is described in U.S. Pat. No. 5,488,504 issued to Terrance L. Worchesky at al in which a hybridized asymmetric Fabry-Perot quantum well light modulator is described. In the subject invention the spatial light modulators are formed by arrays of multiple quantum well gallium arsenide-based devices.
It is a feature of the multiple quantum well device that they can switch as quickly as an electrical signal to them can be changed. This means that the bandwidth is approximately 100 Giga hertz which means that the maximum frame rate is 100 billions of frames per second instead of 10,000 frames per second, which is the best case with liquid crystal based devices. Such devices are typically made of layers of gallium arsenide. Note that the current state of the art with gallium arsenide devices is 300,000 frames per second, where the data rate into the devices from CMOS circuitry is the limiting factor.
The multiple quantum well devices are used in a Van Der Lugt image correlator which is based on a Fourier transform technique that compares converted input images with filters. The filters are created by Fourier transforming reference images and converting them into binary amplitude data. The system operation begins with the image to be identified, i.e. the sample image, being displayed on the input spatial light modulator. The image is first illuminated by a laser beam. Next, the modulated image is reflected onto a Fourier transform lens where it is converted into a Fourier transformed image. The transform image is imaged onto a second spatial light modulator, with the second spatial light modulator providing a Fourier transform of the reference image to be used, with the Fourier transform acting as a filter of the reference image. The identification process involves multiplying the Fourier transform of the input image with the transformed reference image. The output then passes through an inverse Fourier transform lens and is displayed, in one embodiment, on a CCD camera.
A positive correlation appears as bright spot or correlation peak. In one embodiment, a second CCD camera allows an operator to see the input image.
Note that in one embodiment, optical correlation is performed by using reference filters in which the reference filters are Fourier transformed reference images. These filters are designed using amplitude encoded binary phase only principles and are referred to herein as BPOF""s. BPOF""s are used because of their high discrimination capabilities.
Because the multiple quantum well devices can switch at the rate of the drive signal, presently the subject optical image correlator has a functional capability of 300,000 frames per second and is extendable to billions of frames per second.
In summary, a system for performing real time optical comparisons using an optical correlator permits comparing a sampled image to a wide variety of reference images through the utilization of a multiple quantum well spatial light modulator which is utilized to rapidly present a large number of reference images for correlation. The utilization of the multiple quantum well spatial light modulator as the spatial light modulator in a van der Lugt image correlator in combination with a spectrometer permits optical comparisons at 300,000 frames per second versus 10,000 frames per second, the best case for liquid crystal based spatial light modulators.